null
(Ed.)
A promising approach to solving challenging long-horizon tasks has
been to extract behavior priors (skills) by fitting generative models to large offline
datasets of demonstrations. However, such generative models inherit the biases
of the underlying data and result in poor and unusable skills when trained on imperfect demonstration data. To better align skill extraction with human intent we
present Skill Preferences (SkiP), an algorithm that learns a model over human
preferences and uses it to extract human-aligned skills from offline data. After extracting human-preferred skills, SkiP also utilizes human feedback to solve downstream tasks with RL. We show that SkiP enables a simulated kitchen robot to
solve complex multi-step manipulation tasks and substantially outperforms prior
leading RL algorithms with human preferences as well as leading skill extraction
algorithms without human preferences.
more »
« less
Full Text Available